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Applicant's or agent's file reference 
31596-PCT 



International filing date (day/month/year) 
04 March 1999 (04.03.99) 



Priority date (day/month/year) 
04 March 1998 (04.03.98) 



Applicant 

CHANG, Shih-Fu etal 



1 . The designated Office is hereby notified of its election made: 

[")<] in the demand filed with the International Preliminary Examining Authority on: 

22 September 1999 (22.09.99) 



["""] jn a notice effecting later election filed with the International Bureau on: 



2. The election \ X| 

□ 



was not 



made before the expiration of 19 months from the priority date or, where Rule 32 applies, within the time limit under 
Rule 32.2(b). 





Authorized officer 




The International Bureau of W1PO 
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Telephone No.: (41-22) 338.83.38 





Form PCT/IB/331 (July 1992) 




s 



PATENT COOPERATION TREATY 

PCT 

INTERNATIONAL PRELIMINARY EXAMINATION REPORT 
(PCT Article 36 and Rule 70) 



RECD 0 3 JUL 2000 



WIPO 



PCT 



Applicant' s or agent' s file reference 
3 1596- PCT 


FOR FURTHER ACTION See Notification of Transmits oi I ntenmtional 

Preliminary Examination Report (Form PCT/lPEA/410) 


International application No. 
PCT/US99/04776 


International filing date (day/ month /jear) Priority date (day/ month /year) 
04 MARCH 1999 04 MARCH 1998 


International Patent Classification (IPC) or national classification and IPC 
IPC(7): G06F 17/30; 19/00 and US CI.: 707/3,4,46,100,104 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK 



This international preliminary examination report has been prepared by this International Preliminary 
Examining Authority and is transmitted Jfb the applicant according to Article 36. 



This REPORT consists of a total of 

□ 



sheets. 



This report is also accompanied by ANNEXES, i.e., sheets of the description, claims and/or drawings which have 
been amended and are the basis for this report and/or sheets containing rectifications made before this Authority, 
(see Rule 70.16 and Section 607 qf^he Administrative Instructions under the PCT). 



These annexes consist of a total 



607 o£ihi 



sheets. 



3. This report contains indications relating to the following items: 



I J~^J Basis of the report 

II Q Priority 

III Non-establishment of report with regard to novelty, inventive step or industrial applicability 

IV | | Lack of unity of invention 

V [~x| Reasoned statement under Article 35(2) with regard to novelty, inventive step or industrial applicability; 

citations and explanations supporting such statement 

VI | | Certain documents cited 

VII Qj] Certain defects in the international application 

VIII Q^J Certain observations on the international application 



Date of submission of the demand 
22 SEPTEMBER 1 999 


Date of completion of this report 
25 MAY 2000 


Name and mailing address of the IPEA/US 

Commissioner of Patents and Trademarks 
Box PCT 

Washington, D.C. 2023 1 
Facsimile No. (703) 305-3230 


Authorized officer^-""T^ 

Telephone No. (703^308-67 18 r 



Form PCT/IPEA/409 (cover sheet) (July 1998)* 



INTERNATIONAL PRELIMINARY EXAMINATION REPORT 



International application No. 
PCT/US99/04776 



1. Basis of the report 



1 . With regard to the elements of the international application; * 
| x | the international application as originally filed 
the description: 



pages _ 

pages 

pages 



1-12 



NONE 



_ , as originally filed 
filed with the demand 



NONE 



filed with the letter of 



| x| the claims: 

pages 13-16 

pages NONE 

pages NONE 



, as originally filed 

, as. amended (together with any statement) under Article 19 

, filed with the demand 



pages 



NONE 



, filed with the letter of 



| x| the drawings: 



pages 
pages 
pages 



1-3 



NONE 



, as originally filed 

, filed with the demand 



NONE 



the sequence listing part of the description: 

pages NONE 

pages 

pages 



, filed with the letter of , 



NONE 



, as originally filed 

filed with the demand 



NONE 



filed with the letter of . 



2. With regard to the language, all the elements marked above were available or furnished to this Authority in the language in which 
the international application was filed, unless otherwise indicated under this item. 

These elements were available or furnished to this Authority in the following language which is: 

| | the language of a translation furnished for the purposes of international search (under Rule 23.1(b)). 
| | the language of publication of the international application (under Rule 48.3(b)). 

| | the language of the translation furnished for the purposes of international preliminary examination (under Rules 55.2 and/ 
or 55.3). 

3. With regard to any nucleotide and/or amino acid sequence disclosed in the international application, the international 
preliminary examination was carried out on the basis of the sequence listing: 

□ contained in the international application in printed form. 

| | filed together with the international application in computer readable form. 

| | furnished subsequently to this Authority in written form. 

| | furnished subsequently to this Authority in computer readable form. 

□ The statement that the subsequently furnished written sequence listing does not go beyond the disclosure in the 
international application as filed has been furnished. 

I I The statement that the information recorded in computer readable form is identical to the writen sequence listing has 
' — ' been furnished. 

4 | x[ The amendments have resulted in the cancellation of: 

[iD the description, pages NQNE 

the claims, Nos. NONE 

fx] the drawings, sheets/ftg NONE 



5. |x | This report has been drawn as if (some of) the amendments had not been made, since they have been considered to go 

beyond the disclosure as filed, as indicated in the Supplemental Box (Rule 70.2(c)).** 
* Replacement sheets which have been furnished to the receiving Office in response to an invitation under Article 14 are referred to 
in this report as "originally filed" and are not annexed to this report since they do not contain amendments (Rules 70. J 6 
and 70.17). 

**/4/iy replacement sheet containing such amendments must be referred to under item 1 and annexed to this report. 
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International application No. 


INTERNATIONAL PRELIMINARY EXAMINATION REPORT 


PCT/US99/04776 


V. Reasoned statement under Article 35(2) with regard to novelty, inventive step or industrial applicability; 


citations and explanations supporting such statement 




1. statement 




NInvRlty HsTi Claims 1-18 


YES 


Claims 19-24 


IN KJ 


Inventive Step fTS^l Claims NONE 


YES 


Claims 1-24 


NO 


Industrial Applicability (I A) Claims 1-24 


YES 


Claims NONE 


NO 


2. citations and explanations (Rule 70.7) 




Claims 1-24 lack of inventive step under PCT Article 33(3) as being obvious over 


Balogh (US Pat. No. 5,493,677) in view 


of Okamoto (US Pat. No. 5,708,805). 




As per claims 1, 7, 13, and 19 Balogh discloses a method comprising of obtaining at least one initial query for the 


concept (see, col. 15, lines 12-14); 




generating at least one additional query related to the initial query (see, col. 3, lines 4-6); 


producing the additional query for inspection for appropriateness with respect to the concept (see, col. 1 0, lines 23-28); 


displaying the information (see, col. 17, lines 45-46). But, he does not specifically disclose a method of mcluding the 


additional query in the visual template for the concept. However, Okamoto discloses a method of images contain more 


information and are more concrete or real than natural language, many images are employed in various documents (see, col. 


1, lines 11-13). 




It would have been obvious to a person of ordinary skill in the art at the time the Applicant's invention was made to 


modify the teachings of Balogh with including the additional query in the visual template for the concept as disclosed by 


Okamoto, because such modification would provide the system to the degree of similarity with the image retrieving information 


(see, col. 1, lines 54-56). 




As per claims 2, 5, 14, and 17 Balogh discloses a method wherein each query is represented by an icon/example image 


(?ee, figure 14, col. 17, lines 14-16). 




As per claims 3,8,15, and 20 Balogh discloses a method wherein the textual query is obtained from a keyboard (see, 


cols. 13, and 17, lines 43-53, and 2-6). 




As per claims 4, 9, 16, and 21, 




the limitations of claims 4, 9, 16, and 21 have been mentioned in the analysis of claim 1 above. They are, therefore, lack of 


inventive step. 




(Continued on Supplemental Sheet.) 
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INTERNATIONAL PRELIMINARY EXAMINATION REPORT 



International application No. 
PCT/US99/04776 



Supplemental Box 

(To be used when the space in any of the preceding boxes is not sufficient) 



Continuation of: Boxes I - VIII 
I. BASIS OF REPORT: 

5. (Some) amendments are considered to go beyond the disclosure as filed: 
NONE 



Sheet 10 



V. 2. REASONED STATEMENTS - CITATIONS AND EXPLANATIONS (Continued): 

As per claims 6, 10, 18, and 22 Balogh discloses a method wherein appropriateness is ascertained by two-way 
interaction between the system and a user (see, abstract, lines 5-8). 

As per claims 11, and 23 Balogh discloses a method wherein establishing a correspondence between the query and 
the naturel language subset (see, col. 18, lines 38-45); 

labeling different parts of the query as a noun, verb, adjective or preposition (see, col. 14, lines 40-59); 

obtaining clarification if a word in the query is absent from the natural language subset and labeling the word 
accordingly (see, col. 1, lines 9-12). 



As per claims 12, and 24, 

the limitations of claims 12, and 24 have been noted in the analysis of claim 1 above. They are, therefore, lack of inventive 
step. 

NEW CITATIONS 

NONE 
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PATENT COOPERATION TRE/tl'Y 



From the 

INTERNATIONAL PRELIMINARY EXAMINING AUTHORITY 



To: HENRY TANG 

BAKER & BOTTS, LLP 
30 ROCKEFELLER PLAZA 
NEW YORK NY 10112-0228 



PCT 



WRITTEN OPINION 

(PCT Rule 66) 



0^ 

Her 





Date of Mailing t% (k kirrj 1QQQ 

(<ky/ month/ year) |& V IM U V 1333 


Applicant's or agent's file reference 
31596-PCT 


REPLY DUE ... Tvvrk _ fT , c 
within TWO months 

from the above date of mailing 



International application No. 
PCT/US99/04776 



04 MARCH 1999 



International Patent Classification (IPC) or both national classification and IPC 
IPC(6): G06F 17/30; 19/00 and US CI. : 707/3,4,46,100.104 



04 MARCH 1998 



Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK 



1. This written opinion is the 



first 



(first, etc.) drawn by this International Preliminary Examining Authority. 



2. This opinion contains indications relating to the following items: 



1 




II 


□ 


III 


□ 


IV 


□ 


V 




VI 


□ 


VII 


□ 


VIII 


□ 



Lack of unity of invention 

Reasoned statement under Rule 66.2(a)(ii) with regard to novelty, inventive step or industrial applicability; 
citations and explanations supporting such statement 



Docketed 
For / /2000 Bv 



Certain documents cited 

Certain detects in the international application 
VIII | | Certain observations on the international application 

3. The applicant is hereby invited to reply to this opinion. 

When? See the lime limit indicated above. Tha appliumi may, bm'orj ihd aspiration of that timo limit 4 r e qu e ct thiu 

By submitting a written reply, accompanied, where appropriate, by amendments, according to Rule 66.3. 
For the form and the language of the amendments, see Rules 66.8 and 66.9. 

For an additional opportunity to submit amendments, see Rule 66.4. 

For the examiner's obligation to consider amendments and/or arguments, see Rule 66.4 bis. 
For an informal communication with the examiner, see Rule 66.6. 
If no reply is filed, the international preliminary examination report will be established on the basis of this opinion. 

4. The final date bv which the international preliminary 

examination report must be established according to Rule 69.2 is: "4 JULY .uuu = . 



How? 
Also 





Name and mailing address of the IPE.VUS 

Commissioner of PattrnLs and Trail .n.nrks 
Box KT 

Washington. D.C. 20231 
Facsimile No. (703) 305-3230 


Authorized oit'ieer -^J^TT? ' 

WAYNES A.V1SBURY /^^^^^ 
Telephone No. t?03) 305-3828 
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International application No. 


* 


WRITTEN OPINION 








PCT/US99/04776 



I. Basis of the opinion 



1 . This Opinion lias been drawn 011 the basis" of (Substitoie shuts vhhit hm* been famished m :ht nvdnng Ojjicc in response to an 
invitation under Article 14 .n* nfemd to in this opinion as "ongndlfy filed'.)'. 

[ x| the international application as originally filed. 

[x\ the description, pages 1-1- . as originally filed. 

pages NONE . filed with the demand. 

pages NQNE , filed with the letter of 



the claims, 



Nos. 
Nos. 
Nos. 

Nos. 



1-24 



NONE 



NONE 



NONE 



, as originally filed. 

, as amended under Article 19. 

, filed with the demand. 

, filed with the letter of 



the drawings, sheets4+g I' 3 



sheets/fig NONE 



sheeis/fi* NONE 



. as originally filed. 

. filed with the demand. 

. filed with the Jeuer of , 



2. The amendments have resulted in the cancellation of: 
the description, pages NQNE 



the claims, 



Nos 



. NONE 



the drawings, sheets/ft* NONE 



3. Q^J This opinion has been established as if (some oO the amendments had not been made, since they have been 

considered to go beyond the disclosure as filed, us indicated in the S u p pl e m e ntal Box Additional observations below 
(Rule 70.2(c)). 

4. Additional observations, if necessary: 
NONE 



Form PCT/IPEA/408 (Box I) (January !S>94»* 



WRITTEN OPINION 



' International application No. 
I PCT. US99/04776 



j V. Reasoned statement under Rule <i6.2ia)(iij «"iih regard t«* novelty, inventive step or industrial applicability; 
citations and explanations supporting such statement 



1. STATEMENT 
Novelty <N) 



Claims 1- :8 



Claims 19-34 



C!:ii:r.s 
Claims 



NOME 



YES 
NO 

YES 
NO 



Indusinal A-p^abilitv JA Claims J^± 

Chums NONE NO 



2. CITATIONS AND EXPLANATIONS 

Claims 1-24 lack of inventive step under PCT Article 33(3) as being obvious over Balogh (US Pat. No. 5,493,677) in view 
of Okamoto tL'S Pat. No. 5.708.805). 

As per claims 1. 7. 13, ana 19 Balogh discloses a method comprising of obtaining at least one initial query for the 
concept (.see. col. 15. lines 12-14); 

generating at least one additional query related to the irjlial query (see. col. 3. lines 4-6); 

producing me additional query for inspection for appropriateness with respect to the concept (see, col. 10, lines 23-28); 

displaying the information » sec, coi. 17. lines 45-46;. But. he does not specifically disclose a method of including the 
additional .juerv in the visual template :'or the concept. However. Okamoto discloses a method of images contain more 
information and are more concrete or real than natural language, many images are employed in various documents (see, col. 
1. lines 11-13). 

It would have been obvious to a person of ordinary skill in the an at die time the Applicant's invention was made to 
modify the teachings si' Balogh with including the additional q\ier\ in the visual .emplate for the concept as disclosed by 
Okamoto. because such modification ccuid prov ide the system u* '.he degree of similarity with the image retrieving information 

(see. coi. ■. '.ines 5-J--56;. 

As oor claims 2. 5. 14. and 7 Baiogh discloses a method w herein each query :s represented by an icon/example image 
(see. figure 14. coi. P. lines 14-16 . 

As ;x*r claims 3, S. 15, and 20 Ba.ogh discloses a method wherein the :extui; query is obtained from a keyboard (see. 
cols. 13. and 17. \'uk\ 43-53. and 2-6). 

As per claims 4. 9. 16, ar.d 21. 
the limitations of cia:ms 4. 9. 16, ../id -1 nave been mentioned i;i :he analysis of cia:m 1 above. They are, therefore, lack of 
inventive step. 

(Continued on Supplemental Sheet.* 
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WRITT1 N OPINION 


International application No. 
PCT/US99/04776 




Supplemental Box 

(To be used w hen the space in any of the preceding boxes is no; sufficient) 




Continuation of: Boxes I - VIII 


Sheet 10 



TIME LIMIT: 

The time limit set for response to a Written Opinion may not be extended. 37 CFR 1.484(d). Any response 
received after the expiration of the tune limit set in the Written Opinion will not be considered in preparing the International 
Preliminary Examination Report. 

V. 2. REASONED STATEMENTS - CITATIONS AND EXPLANATIONS (Continued): 

As per claims 6. 10, 18, am! 22 Balogh discloses a method wherein appropriateness is ascertained by two-way 
interaction between the system and a user (see, abstract, lines 5-8). 

As per claims 1 1, and 23 Balogh discloses a method wherein establishing a correspondence between the query and 
the nature! language subset (see. col. 18. lines 38-45); 

labeling different parts of the query as a noun, verb, adjective or preposition (see. col. 14. lines 40-59); 

obtaining clarification if a word in the query is absent from the natural language subset and labeling the word 
accordingly (see. col. 1. lines 9-12). 



As per claims 12. and 24, 

the limitations of claims 12. and 24 have been noted in the analysis of claim I above. They are, therefore, lack of inventive 
step. 



- NEW CITATIONS 

NONE 



Fonn PCT/lPEA/408 (Supplemental Box) (January 1994) ★ 



INTERNATIONAL SEARCH REPORT 



International application No. 
PCT/US99/04776 



A. CLASSIFICATION OF SUBJECT MATTER 
IPC(6) :G06F 17/30; 19/00 

US CL : 707/3,4,46,100.104 
According co I nternational Patent Classification (IPC) or to both national classification an d IPC 

B. FIELDS SEARCHED " " 



i Minimum documentation searched (classification system followed by classification symbols) 
U.S. : 707/3.4,46,100,104 



Documentation searched other than minimum documentation to the extent that such documents 



are included in the fields searched 



I Electronic da* base consulted during the international search (name of data base and. where practicable, scan* terms used) 



APS, IS&R, EIC 

SEARCH TERMS: VISUAL, IMAGE, NATURAL LANGUAGE. RETRIEVE, SVT 



I C DOCUMENTS CONSIDERED TO BE RELEVANT 



Category* 

Ix 



Citation of document, with indication, where appropriate, of the relevant passages 

Tt 493,677 A (BALOGH et *!•> 20 February 1996, col. 14, lines 



US 5,724,484 A (KAGAMI et al.) 03 March 1998, col. 8, lines 47- 

60. 

US 5 708,805 A (OKAMOTO et al.) 13 January 1998, col. 1, lines 

30-49. 



□ 

Further documents are listed in the continuation of Box C. | J 



Relevant to claim No. 

1-24 

1-24 
1-24 



*A" 

"E" 
■L" 

•O" 

Date 



Special categoric i of cited documents: 

document defining the general state of the art which is not considered 
to be of particular relevance 

earher document published on or after the intemauonai filing date " X " 

document which may throw doubts on priority claim(s) or which is 
cited to establish the publication date of another ciiauon or other 
special reason us specified) ■ y 

document referring to an oral disclosure, use, exhibition or other 
means 

document published prior to the intemauonai filing date but later than - A . 

the priority date clatmed & 

of the actual completion of the international search 



See patent family annex. 



29 APRIL 1999 

Name and mailing address of the ISA/ US 
Commissioner of Patents and Trademarks 
Box PCT 

Washington. D.C. 20231 
Facsimile No. (703) 308-5403 
Form PCT/ISA/210 (second sheet)(July 1992) i 



later document published after the intemauonai filing date or priority 
date and not in contact with the application but cited to understand 
tne principle or theory underlying the invention 

document of parucjiar relevance, the claimed invention cannot be 
considered novel or cannot be considered to involve an inventive step 
when the document is taken alone 

document of particular relevance; the claimed invention cannot be 
considered to mvolve an in v enure step when the document is 
combined with one or more other such documents, such combination 
being obvious to a person skilled in the art 

document member of the same patent family 

Date of mailing of the international search report 



14 MAY 1999 



Authorized officer 

PAUL R. LINTZ 
Telephone No. (703) 305y^832 ^ 



PATENT COOPERATION TREATY 



From the 

INTERNATIONAL PREUMJJM. 



/ EXAMINING AUTHOR* i 



To: 



HENRY TANG 
BAKER & BOTTS. LLP 
30 ROCKEFELLER PLAZA 
NEW YORK NY 10112-0228 



NOTIFICATION-OF4l£GEIPT— ~ Q 
OF DEMAND BY COMPETEOTIhrreRNATIONAL ^ 
PRELIMINARY EXAMINING AUTHORITY ^J[A 

(PCT Rule 593(e) and 61.1(b), first sentence 
and Administrative Instructions, Section 601( 



Date of mailing 
(dayfatorth/yeai) 




12 OCT 1999 



Applicant's or agent's file reference 

31596- PCT 


IMPORTANT NOTIFICATION 


International application No. 

PCT/US99/04776 


International filing date (day/month/year) 

04 MAR 99 


Priority date (day /month/year) 

04 MAR 98 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF 
NEW YORK 



1 . The applicant is hereby notified that this International Preliminary Examining Authority considers die following date as the 
date of receipt of the demand for international preliminary examination of the international application: 



2. That date of reeeipt is: 

Y~pf the actual date of receipt of the demand by this Authority (Rule 61.1(b)). 

[ [ the actual date of receipt of the demand on behalf of this Authority (Rule 59.3(e)). 



3. 



□ 



| [ the date on which this Authority has, in response to the invitation to correct defects in the demand (Form 
PCT/IPEA/404) , received the required corrections. 



ATTENTION: That date of receipt is AFTER the expiration of 19 months from the priority date. Consequently, the 
election(s) made in the demand does (do) not have the effect of postponing the entry into the national phase until 
30 months from the priority date (or later in some Offices) (Article 39(1)). Therefore, the acts for entry into xhe 
national phase must be performed within 20 months from the priority date (or later in some Offices) (Article 22). 
For details, see the PCT Applicant's Guide, Volume n. 



□ 



(If applicable) This notification confirms the information given by telephone, facsimile transmission or in person 
on: 



4. Only where paragraph 3 applies, a copy of this notification has been sent to the International Bureau. 



i 





Name and mailing address of the IPEA/US 
Assistant Commissioner for Patents 
Box PCT 

Washington, D.C. 20231 Attn: IPEA/US 
Facsimile No. 


Authorized officer / 

TyettaYounfl >/ // 
PCTAnteman Appl PluutftfRQ CT» Z^f 

*M^3<571 ^ (j 



Form PCT/IPEA/402 (July 1998) 



PATENT COOPERATION TREATY 

From the INTERNATIONAL BUREAU 



WO 99/45483 
PCT/US99/04776 



PCT 

NOTICE INFORMING THE APPLICANT OF THE 
COMMUNICATION OF THE INTERNATIONAL 
APPLICATION TO THE DESIGNATED OFFICES 

(PCT Rule 47.1(c), first sentence) 



To: 



TANG, Henry 

Baker & Botts, LLP - 

30 Rockefeller Plaza . " ' \_ p 

New York, NY 101 12-0228 

ETATS-UNIS D'AMERIQyE C - D o d ; [2: 03 



Date of mailing (day/month/year) 

10 September 1999 (10.09.99) 






Applicant's or agent's file reference 
31596-PCT 


IMPORTANT NOTICE 


International application No. 
PCT/US99/04776 


International filing date (day/month/year) 
04 March 1999 (04.03.99) 


Priority date (day/month/year) 
04 March 1998 (04.03.98) 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK et al 



1 . Notice is hereby given that the International Bureau has communicated, as provided in Article 20, the international application 
to the following designated Offices on the date indicated above as the date of mailing of this Notice: 

EP,JP,KR,US 



In accordance with Rule 47.1 (c), third sentence, those Offices will accept the present Notice as conclusive evidence that 
the communication of the international application has duly taken place on the date of mailing indicated above and no copy 
of the international application is required to be furnished by the applicant to the designated Office(s). 

2. The following designated Offices have waived the requirement for such a communication at this time: 
CA 



The communication will be made to those Offices only upon their request. Furthermore, those Offices do not require the 
applicant to furnish a copy of the international application (Rule 49.1 (a-bis)). 

3. Enclosed with this Notice is a copy of the international application as published by the International Bureau on 
10 September 1999 (10.09.99) under No. WO 99/45483 



REMINDER REGARDING CHAPTER II (Article 31{2)(a) and Rule 54.2) 

If the applicant wishes to postpone entry into the national phase until 30 months (or later in some Offices) from the priority 
date, a demand for international preliminary examination must be filed with the competent International Preliminary 
Examining Authority before the expiration of 19 months from the priority date. 

It is the applicant's sole responsibility to monitor the 19-month time limit. 

Note that only an applicant who is a national or resident of a PCT Contracting State which is bound by Chapter II has the 
right to file a demand for international preliminary examination. 



REMINDER REGARDING ENTRY INTO THE NATIONAL PHASE (Article 22 or 39(1)) 

If the applicant wishes to proceed with the international application in the national phase, he must within 20 months 
or 30 months, or later in some Offices, perform the acts referred to therein before each designated or elected Office. 

For further important information on the time limits and acts to be performed for entering^he^iatfoh"aTphase, see the 
Annex to Form PCT/IB/301 (Notification of Receipt of Record Copy) and Volume II of the PCT'Applicant's Guide. 



DOCKETED 



/1999 Bl 



The International Bureau of WIPO 
34, chemin des Colombettes 
1211 Geneva 20, Switzerland 



Facsimile No. (41-22) 740.14.35 



Authorized officer 

J. Zahra 

Telephone No. (41-22) 338.83.38 



Form PCT/1B/308 (July 1996) 



2829100 



PATENT COOPERATION TREATY 



From the INTERNATIONAL BUREAU 



PCT 



NOTIFICATION OF RECEIPT OF 
RECORD COPY 

(PCT Rule 24.2(a)) 



To: 



TANG, Henry 
Baker & Botts, LLP 
30 Rockefeller Plaza 
New York, NY 10112-0228 
ETATS-UNIS D'AMERIQUE 



IVED 



.L.P. 

qqiluV in f«HBf9» 



Date of mailing (day/month/year) 
26 April 1999 (26.04.99) 



IMPORTANT NOTIFICATION 
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The applicant is hereby notified that the International Bureau has received the record copy of the international application as 
detailed below. 

Name(s) of the applicant(s) and State(s) for which they are applicants: 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK (for all designated 
States except SU) 

CHANG, Shih-Fu et al (for US) 
International filing date 04 March 1 999 (04.03.99) 
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ATTENTION 

The applicant should carefully check the data appearing in this Notification. In case of any discrepancy between these data 
and the indications in the international application, the applicant should immediately inform the International Bureau. 

In addition, the applicant's attention is drawn to the information contained in the Annex relating to: 



time limits for entry into the national phase 
confirmation of precautionary designations 
requirements regarding priority documents 



A copy of this Notification is being sent to the receiving Office and to the International Searching Authority. 
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The International Bureau of WIPO 
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INFORMATION ON TIME LIMITS FOR ENTERING THE NATIONAL PHASE 

The applicant is reminded that the "national phase" must be entered before each of the designated Offices indicated in the 
Notification of Receipt of Record Copy (Form PCT/IB/301) by paying national fees and furnishing translations, as prescribed by 
the applicable national laws. 

The time limit for performing these procedural acts is 20 MONTHS from the priority date or, for those designated States 
which the applicant elects in a demand for international preliminary examination or in a later election, 30 MONTHS from the 
priority date, provided that the election is made before the expiration of 19 months from the priority date. Some designated (or 
elected) Offices have fixed time limits which expire even later than 20 or 30 months from the priority date. In other Offices an 
extension of time or grace period, in some cases upon payment of an additional fee, is available. 

In addition to these procedural acts, the applicant may also have to comply with other special requirements applicable in 
certain Offices. It is the applicant's responsibility to ensure that the necessary steps to enter the national phase are taken in a 
timely fashion. Most designated Offices do not issue reminders to applicants in connection with the entry into the national 
phase. 

For detailed information about the procedural acts to be performed to enter the national phase before each designated 
Office, the applicable time limits and possible extensions of time or grace periods, and any other requirements, seethe relevant 
Chapters of Volume II of the PCT Applicant's Guide. Information about the requirements for filing a demand for international 
preliminary examination is set out in Chapter IX of Volume I of the PCT Applicant's Guide. 

GR and ES became bound by PCT Chapter II on 7 September 1996 and 6 September 1997, respectively, and may, therefore, 
be elected in a demand or a later- election filed on or after 7 September 1996 and 6 September 1997, respectively, regardless of 
the filing date of the international application. (See second paragraph above.) 

Note that only an applicant who is a national or resident of a PCT Contracting State which is bound by Chapter II has 
the right to file a demand for international preliminary examination. 

CONFIRMATION OF PRECAUTIONARY DESIGNATIONS 

This notification lists only specific designations made under Rule 4.9(a) in the request, tt is important to check that these 
designations are correct. Errors in designations can be corrected where precautionary designations have been made under 
Rule 4.9(b). The applicant is hereby reminded that any precautionary designations may be confirmed according to Rule 4.9(c) 
before the expiration of 15 months from the priority date. If it is not confirmed, it will automatically be regarded as withdrawn 
by the applicant. There will be no reminder and no invitation. Confirmation of a designation consists of the filing of a notice 
specifying the designated State concerned (with an indication of the kind of protection or treatment desired) and the payment 
of the designation and confirmation fees. Confirmation must reach the receiving Office within the 15-month time limit. 

REQUIREMENTS REGARDING PRIORITY DOCUMENTS 

For applicants who have not yet complied with the requirements regarding priority documents, the following is recalled. 

Where the priority of an earlier national, regional or international application is claimed, the applicant must submit a copy 
of the said earlier application, certified by the authority with which it was filed ("the priority document") to the receiving Office 
(which will transmit it to the International Bureau) or directly to the International Bureau, before the expiration of 16 months from 
the priority date, provided that any such priority document may still be submitted to the International Bureau before that date of 
international publication of the international application, in which case that document will be considered to have been received 
by the International Bureau on the last day of the 16-month time limit (Rule 17.1(a)). 

Where the priority document is issued by the receiving Office, the applicant may, instead of submitting the priority 
document request the receiving Office to prepare and transmit the priority document to the International Bureau. Such request 
must be made before the expiration of the 16-month time limit and may be subjected by the receiving Office to the payment 
of a fee (Rule 1 7.1 (b)). 

If the priority document concerned is not submitted to the international Bureau or if the request to the receiving Office 
to prepare and transmit the priority document has not been made (and the corresponding fee, if any, paid) within the applicable 
time limit indicated under the preceding paragraphs, any designated State may disregard the priority claim, provided that no 
designated Office may disregard the priority claim concerned before giving the applicant an opportunity to furnish the priority 
document within a time limit which is reasonable under the circumstances. 

Where several priorities are claimed, the priority date to be considered for the purposes of computing the 16-month time 
limit is the filing date of the earliest application whose priority is claimed. 
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under Article 3 1 of the Patent Cooperation Treaty: 
The undersigned requests that the international application specified below be the subject of 
international preliminary examination according to the Patent Cooperation Treaty and 
hereby elects all eligible States (except where otherwise indicated). 
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International application No. 
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(Earliest) Priority date (day/month/year) 
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State (that is, country) of nationality: 
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name of country.) 



CHANG, SHIH-FU 
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TW 
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US 
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US 
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Name and address: (Family name followed by given name; for a legal entity, full official designation. The address must include postal code and 
name of country.) 



State (that is, country) of nationality: 



State (that is, country) of residence: 



Name and address: (Family name followed by given name; for a legal entity, full official designation. The address must include postal code and 
name of country.) 



State (that is, country) of nationality: 



State (that is, country) of residence: 
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name of country.) 
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□ is hereby appointed, specifically for the procedure before the International Preliminary Examining Authority, in 
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TANG, HENRY and 

BUSINGER, PETER A. 

Baker & Botts, LLP 
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□ Address for correspondence: Mark this check-box where no agent or common representative is/has been appointed and 
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Box No. IV BASIS FOR INTERNATIONAL PRELIMINARY EXAMINATION 



Statement concerning amendments:* 

1. The applicant wishes the international preliminary examination to start on the basis of: 

the international application as originally filed, 
the description I I as originally filed 

I | as amended under Article 34 

the claims I I as originally filed 

[ | as amended under Article 19 (together with any accompanying statement) 
| | as amended under Article 34 

the drawings I I as originally filed 

1 1 as amended under Article 34 

2. I I The applicant wishes any amendment to the claims under Article 1 9 to be considered as reversed. 

3. | | The applicant wishes the start of the international preliminary examination to be postponed until the expiration of 

20 months from the priority date unless the International Preliminary Examing Authority receives a copy of any 
amendments made under Article 1 9 or a notice from the applicant that he does not wish to make such amendments 
(Rule 69. 1(d)). (This check-box may be marked only where the time limit under Article 19 has not yet expired.) 
* Where no check-box is marked, international preliminary examination will start on the basis of the international application as 
originally filed or, where a copy of amendments to the claims under Article 19 and/or amendments of the international 
application under Article 34 are received by the International Preliminary Examining Authority before it has begun to draw up 
a written opinion or the international preliminary examination report, as so amended. 



Language for the purposes of international preliminary examination: English 
[)Q which is the language in which the international application was filed. 
I I which is the language of a translation furnished for the purposes of international search. 
I 1 which is the language of publication of the international application. 

I I which is the language of the translation (to be) furnished for the purposes of international preliminary examination. 
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The applicant hereby elects all eligible States (that is, all States which have been designated and which are bound by Chapter II of the 
PCT) 

excluding the following States which the applicant wishes not to elect: 
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Examining Authority use only 

received not received 


1 . translation of international application 


sheets 


□ 


□ 


2. amendments under Article 34 


sheets 


□ 


□ 


3. copy (or where required, translation) 
of amendments under Article 19 


sheets 


□ 


□ 


4. copy (or, where required, translation) 
of statement under Article 1 9 


sheets 


□ 


□ 


5. letter 


sheets 


□ 


□ 


6. other (specify) 


sheets 


□ 


□ 
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1 . [^| fee calculation sheet 

2. | | separate signed power of attorney 

3 I I copy of general power of attorney; 
' ' reference number, if any: 



4. | [ statement explaining lack of signature 

5. I I nucleotide and or amino acid sequence listing in 
L— J computer readable form 

6. ^| other (specify) : Transmittal Letter 



Box No. Vn SIGNATURE OF APPLICANT, AGENT OR COMMON REPRESENTATIVE 



Next to each signature, indicate the name of the person signing and the capacity in which the person signs (if such capacity is not 
obvious from reading the demand). 
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2. Adjusted date of receipt of demand due 
to CORRECTIONS under Rule 60.1(b): 



3. I I The date of receipt of the demand is AFTER the expiration of 19 months | I The applicant has been 

' from the priority date and item 4 or 5, below, does not apply. "—J informed accordingly. 



4. I I The date of receipt of the demand is WITHIN the period of 1 9 months from the priority date as extended by virtue of 
■ — 1 Rule 80.5. 



5. I I Although the date of receipt of the demand is after the expiration of 1 9 months from the priority date, the delay in arrival is 
1 — 1 EXCUSED pursuant to Rule 82. 
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Calculation of prescribed fees 



1. Preliminary examination fee 
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handling fee.) 

3. Total of prescribed fees 

Add the amounts entered at P and H 

and enter total in the TOTAL box 



490.00 



162.00 H 
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TOTAL 



Mode of Payment 

□ authorization to charge deposit 
account with the IPEA (see below) 

cheque 
| | postal money order 
| | bank draft 



| | cash 

| | revenue stamps 

[ | coupons 

| I other (specify): 



Deposit Account Authorization (this mode of payment may not be available at all IPEAs) 

The IPEA/ US [ | is hereby authorized to charge the total fees indicated above to my deposit account. 

(this check-box may be marked only if the conditions for deposit accounts of the IPEA so permit) is 
hereby authorized to charge any deficiency or credit any overpayment in the total fees indicated 
above to my deposit account. 



02-4377 



22 September 1999 



Deposit Account Number 



Date (day/month/year) 
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(57) Abstract 

For database image/video retrieval, a semantic visual template (SVT) is a set of icons of example scenes/objects that characterize a 
concept, e.g. skiing, sunset and the like. SVTs provide for two-way interaction between a user and a system. The user can provide the 
system with an initial sketch or example image, as a seed to the system to automatically generate other representations of the same concept 
The user then can pick those views for inclusion that are plausible for representing the concept. When an SVT has been established, the 
database can be searched with it, for the user to provide relevancy feedback on the returned results. With established SVTs, the user can 
interact with the system at concept level. In forming new concepts, pre-existing SVTs can be used. A limited vocabulary can be parsed, 
in conjunction with semantic visual templates for querying the system. 
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For database image/video retrieval, a semantic visual template (SVT) is a set of icons of example scenes/objects that characterize a 
concept, e.g. skiing, sunset and the like. SVTs provide for two-way interaction between a user and a system. The user can provide the 
system with an initial sketch or example image, as a seed to the system to automatically generate other representations of the same concept 
The user then can pick those views for inclusion that are plausible for representing the concept. When an SVT has been established, the 
database can be searched with it, for the user to provide relevancy feedback on the returned results. With established SVTs, the user can 
interact with the system at concept level. In forming new concepts, pre-existing SVTs can be used. A limited vocabulary can be parsed, 
in conjunction with semantic visual templates for querying the system. 
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METHOD AND SYSTEM FOR GENERATING SEMANTIC 
VISUAL TEMPLATES FOR IMAGE AND VIDEO RETRIEVAL 

Technical Field 

The invention relates to database still image, video and audio retrieval and, 
more particularly, to techniques which facilitate access to database items. 

Background of the Invention 
5 Increasingly, as images and videos are produced, disseminated and stored in 

digital form, tools and systems for searching and retrieving visual information are 
becoming important. However, while efficient "search engines" have become widely 
available for text data, corresponding tools for searching for visual image and video 
data have remained elusive. 

1 0 Typically, in available image and video databases, keyword techniques are 

used to index and retrieve images. Known retrieval systems of this type suffer from a 
number of drawbacks, as images may not be associated with textual information, as 
manual inclusion of captions is time consuming and subjective, and as textual 
annotations are usually extrinsic, failing to represent the intrinsic visual 

1 5 characteristics of a scene. For example, a textual description such as "A man stands 
against a brick wall" conveys little visual information about either the man or the 
brick wall. Such visual information is often vital in retrieving a particular video. 

Recently, researchers have begun to explore new forms of image and video 
repository retrieval. Such exploration is based on similarity of visual attributes, e.g. 

20 color, texture, shape and spatial and temporal relationships amongst the objects that 
make up a video. In this paradigm, queries are usually specified by giving an 
example image or a visual sketch. Then, retrieval systems return the image or video 
that have the highest similarity to the given example or sketch. 

Summary of the Invention 

25 For ease of retrieval of images and videos from a database, the database can be 

indexed using a collection of visual templates. Preferably, in accordance with an 
aspect of the invention, the visual templates represent semantic concepts or 
categories, e.g. skiing, sunset and the like. There results an architecture using 
semantic visual templates (SVT), with each SVT standing for a concept and 

30 consisting of queries that describe the concept well. 

Semantic visual templates can be established by an interactive process 
between a user and a system. The user can provide the system with an initial sketch 
or example image, as a seed to the system to automatically generate other 
representations of the same concept. The user then can pick those views for inclusion 
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that are plausible for representing the concept. When an SVT has been established, 
the database can be searched with it, for the user to provide relevancy feedback on the 
returned results. With established SVT's , the user can interact with the system at 
concept level. In forming new concepts, pre-existing SVT's can be used.. 
5 Provided further is a technique for parsing a limited vocabulary of 

words in conjunction with semantic visual templates for querying the system. 

Brief Description of the Drawing 

Fig. 1 is a schematic of an interactive technique for generating a library or 
collection of semantic visual templates in accordance with a preferred embodiment of 
1 0 the invention. 

Fig. 2 is a diagram which illustrates a concept having necessary and sufficient 
conditions. 

Fig. 3 is a diagram which illustrates query generation. 
Fig. 4 is a schematic of an interactive system in accordance with a preferred 
15 further embodiment of the invention, including audio processing. 

Fig. 5 shows a set of icons exemplifying the concept "high jump". 
Fig. 6 shows a set of icons exemplifying the concept "sunset". 
Fig. 7 shows a semantic visual template for the concept "slalom" 

20 Detailed Description 

Incorporated herein by reference are U.S. provisional patent application 
No. 60/045,637, filed May 5, 1997 and PCT International Application 
No. PCT/US98/09124 filed May 5, 1998 wherein Canada, Japan, the Republic of 
Korea and the United States of America are designated, describing techniques for 

25 object-based spatial and temporal visual searching using visual templates to search for 
images and video in different categories. Such search techniques, referred to as 
VideoQ, can be used in conjunction with the present invention. 

In a video stream, the beginning and end of a scene can be determined using 
VideoQ, for example, which technique can be used further to compensate for camera 

30 motion for extracting objects in the scene. Still using VideoQ, each object can be 
characterized by salient attributes such as color, texture, size, shape and motion, for 
example. There results a video object database consisting of all the objects extracted 
from the scene and their attributes. 



35 



Visual Templates 

A visual template represents an idea, in the form of a sketch or an animated 
sketch. As a single visual template may be a poor representative of a class of interest, 
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a library of visual templates can be assembled, containing representative templates for 
different semantic classes. For example, when searching for video clips of the class 
Sunset, one could select one or more visual templates corresponding to the class and 
use similarity-based querying to find video clips of sunsets. 

An important advantage of using a visual template library lies in linkage of a 
low-level visual feature representation to high-level semantic concepts. For example, 
if a user enters a query in a constrained natural language form as described in the 
above-referenced patent applications, visual templates can be used to transform the 
natural language query into automated queries specified by visual attributes and 
constraints. When visual content in the repository or database is not indexed 
textually, customary textual search methods cannot be applied directly. 

Semantic Visual Templates TSVT) 

A semantic visual template is the set of visual templates associated with a 
particular semantic. This notion of an SVT has certain key properties as follows: 

Semantic visual templates are general in nature. For a given concept, there 
should be a set of visual templates that cover that concept well. Examples of 
successful SVT's are Sunset, High Jump, Down-hill Skiing. 

A semantic visual template for a concept should be small but cover a large 
percentage of relevant images and videos in the collection, for high precision-recall 
performance. 

Our procedures for finding semantic visual templates for different concepts 
are systematic, efficient, and robust. Efficiency refers to the convergence to a small 
visual template set. Robustness is demonstrated by applying the new library of 
templates to new image and video collections. 

With reference to VideoQ, a semantic visual template can be understood 
further as a set of icons or example scenes/objects that represent the semantic with 
which the template is associated. From a semantic visual template, feature vectors 
can be extracted for querying. The icons are animated sketches. In VideoQ, the 
features associated with each object and their spatial and temporal relationships are 
important. Histograms, texture and structural information are examples of global 
features that can be part of such a template. The choice between an icon-based 
realization versus a feature vector set formed out of global characteristics depends 
upon the semantic to be represented. For example, a sunset scene may be adequately 
represented by a couple of objects, while a waterfall or a crowd is better represented 
using a global feature set. Hence, each template contains multiple icons, example 
scenes/objects to represent a concept. The elements of the set can overlap in their 
coverage. Desirably, coverage is maximized with a minimal template set. 
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Each icon for a concept, e.g. down-hill ski, sunset, beach crowd, is a visual 
representation consisting of graphic objects resembling the actual objects in a scene. 
Each object is associated with a set of visual attributes, e.g. color, shape, texture, 
motion. The relevancy of each attribute and each object to the concept is also 
5 specified. For example, for "sunset", color and spatial structures of the objects such 
as sun and sky are more relevant. For a sunset scene, the sun object may be optional, 
as there may be sunset videos in which the sun is not visible. For the concept "high 
jump", the motion attribute of the foreground object is mandatory, the texture 
attribute of the background is non-mandatory, and both are more relevant than other 
1 0 attributes. Some concepts may need just one object to represent the global attributes 
of the scene. 

Fig. 5 shows several potential icons for "high jump", and Fig. 6 for "sunset". 
The optimal set of icons should be chosen based on relevancy feedback and maximal 
coverage in terms of recall as described below in further detail. 

1 5 We have devised efficient techniques for generating semantic visual templates 

for various concepts. Each semantic concept may have a few representative visual 
templates, which can be used to retrieve a significant portion of images and video, for 
positive coverage or high recall from the repository. The positive coverage sets for 
different visual templates may overlap. Therefore, it is an objective to find a small set 

20 of visual templates with large, minimally overlapping positive coverage. 

Users may provide initial conditions for effective visual templates. For 
example, a user may use a yellow circle (foreground) and a light-red rectangle 
(background) as an initial template for retrieving sunset scenes. Also, users may 
indicate weights and relevancy of different objects, attributes, and necessary 

25 conditions pertaining to the context by answering an interactive questionnaire. The 
questionnaire is sensitive to the current query that the user has sketched out on a 
sketchpad, for example. 

Given the initial visual template and relevancy of all visual attributes in the 
template, the search system will return a set of most similar images/video to the user. 

30 Given the returned results, the user can provide subjective evaluation of the returned 
results. The, precision of the results and positive coverage, i.e. recall can be 
computed. 

The system can determine an optimal strategy for altering the initial visual 
query and generate modified queries based on: 

35 1 • The relevancy factor of each visual attribute obtained by the user 

questionnaire, 

2. Precision-recall performance of the previous query, 

and 
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3. Information about feature level distribution of images and video in the 
repository. 

Such features are embodied in a technique as conceptually exemplified by 
Fig. 1, with specific illustration of a query for the concept "high jump". The query 
5 includes three objects, namely two stationary rectangular background fields and an 
object which moves to the lower right. For each object in the query, four qualities are 
specified with associated weights, e.g. color, texture, shape and size, represented in 
Fig. 1 by vertical bars. A new query can be formed by stepping at least one of the 
qualities, at which point user interaction can be invoked for deciding as to plausibility 
1 0 for inclusion as an icon in the template. Once a suitable number of icons have been 
assembled into a tentative template, this template can be used for a database search. 
The results of the search can be evaluated for recall and precision. If acceptable, the 
template can be stored as a semantic visual template for "high jump". 

Template Metric 

1 5 The fundamental video data unit may be termed a video shot, comprising 

multiple segmented video objects. The lifetime of any particular video object may be 
equal to or less than the duration of the video shot. A similarity measure D between a 
member of the SVT set and a video shot can be defined as 

D = min { G> f • £,„ d^, 0' s ) + o s • d s } (l) 

20 where the Oj are the objects specified in the template, O'j are the matched objects for 
Oi.dfis the feature distance between its arguments, d,, is the similarity between the 
spatial-temporal structure in the template and that among matched objects in the 
video shot, co f and co s are the normalized weights for the feature distance and the 
structure dissimilarity. The query procedure is to generate a candidate list for each 

25 object in the query. Then, the distance D is the minimum over all possible sets of 
matched objects that satisfy the spatial-temporal restrictions. For example, if the 
semantic template has three objects and two candidate objects are kept for each single 
object query, there will be at most eight potential candidate sets of objects considered 
in computing the minimal distance in Equation 1. 

30 Given N objects in the query, this appears to require searching over all sets of 

N objects that appear together in a video shot. However, for computational economy, 
the following more economical procedure can be adopted: 

1 . Each video object, O f , say, is used to query the entire object database, 
resulting in a list of matched objects which can be kept short by using a threshold. 

35 Only objects included in this list are then considered as candidate objects matching 
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2. The candidate objects on the list are then joined, resulting in the final set of 
matched objects on which the spatial-temporal structure relationships will be verified. 

Template peneratinn 

5 Two-way interaction is used between a user and the system for generating the 

templates. Given the initial scenario and using relevancy feedback, the technique 
converges on a small set of icons that gives maximum recall. A user furnishes an 
initial query as a sketch of the concept for which a template is to be generated, 
consisting of objects with spatial and temporal constraints. The user can also specify 

1 0 whether the object is mandatory. Each object has features to which the user assigns 
relevancy weights. 

The initial query can be regarded as a point in a high-dimensional feature 
space into which all videos in the database can be mapped. For automatic generation 
of the set of test icons it will be necessary to make jumps in each of the features of 

1 5 each of the objects, after quantizing the space. For quantizing, a step size can be 
determined with the help of the weight that the user has specified along with the 
initial query, which weight can be regarded as a measure for the degree of relevancy 
attributed by the user to the feature of the object.. Accordingly, a low weight results 
in coarse quantization and vice versa, e.g. when 

20 A(w) = l/(a-G> + b) (2) 

where A is the jump distance corresponding to a feature, w is the weight associated 
with the feature, and a and b are parameters which are chosen such that A(0) = 1 and 
A(l) = do which is a system parameter related to thresholding, set at 0.2 in a prototype 
system. Using the jump distance, the feature pace is quantized into hyper-rectangles. 

25 For example, for color the cuboids can ge generated using the metric for the LUV 
space along with A(co). 

To prevent the total number of possible icons from increasing rapidly, joint 
variation of the features is prevented, e.g. as follows: 

1 . For each feature in the object, the user picks a plausible set for that feature. 

30 2. The system then performs a join on the set of features associated with the 

object. 

3. The user then picks the joins that most likely represent variations of the 
object, resulting in a candidate icon list. 

In a multiple object case, an additional join can be included in step 2 with 
35 respect to the candidate lists for each object. Once a list of plausible scenarios has 
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been generated, the system is queried using the icons which the user has picked. 
Using relevancy feedback on the returned results, with the user labeling the returned 
results as positive or negative, those icons are determined which result in maximum 
recall. 

Concept Covers 

Sufficiently many "coverings" are desired of a concept, for a user to search a 
database. Each covering can reside in a different feature space, e.g. a "sunset" may be 
described at the object level as well as the global level. A global-level description 
may take the form of a color or texture histogram. An object-level description may be 
a collection of two objects such as the sky and the sun. These objects may be further 
quantified using feature level descriptors. 

As illustrated by Fig. 2, a concept (e.g. Sunset) has two different kinds of 
conditions, necessary (N) and sufficient (S). A semantic visual template is a 
sufficient condition on the concept and not a necessary one, so that a particular S VT 
need not cover the concept to its full extent. Additional templates may be generated 
manually i.e. as the user inputs additional queries. The task is undertaken for each 
concept. Necessary conditions can be imposed on a concept, thereby automatically 
generating additional templates, given an initial query template. 

The user interacts with the system through a "concept questionnaire", to 
specify necessary conditions for the semantic searched for. These conditions may 
also be global, e.g. the global color distribution, the relative spatial and temporal 
interrelationships etc. Once the necessary and sufficient conditions for the concept 
are established, the system moves in the feature space to generate additional 
templates, with the user's original one as a starting point. This generation is also 
modified by the relevancy feedback given to the system by the user. By analyzing the 
relevancy feedback, new rules can be determined pertaining to the necessary 
conditions. These can be used further to modify the template generation procedure. 
The rules are generated by looking at the correlation between the conditions deemed 
necessary for a concept with the videos that have been marked as relevant by the user. 
This principle of determining rules (or implications) is akin to the techniques found in 
"data mining" as described in the above-identified patent applications and in papers 
by S. Brin et al., 'Dynamic Itemset Counting and Implication Rules for Market 
Basket Data", ACM SIGMOD Conference on Management of Data, 1997, pp. 255- 
246 and S. Brin et al., "Beyond Market Baskets: Generalizing Association Rules to 
Correlations", ACM SIGMOD Conference on Management of Data, 1997, pp. 265- 
276. 
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Rule Generation Examp le 

A query, for a "crowd of people", in VideoQ is in the form of a sketch. The 
user has specified a visual query with an object, giving weights for color and size, but 
is unable to specify a more detailed description in the form of either texture (of the 
crowd) or the relative spatial and temporal movements that characterize the concept 
of a crowd of people. However, since he feels that the idea of a "crowd" is strongly 
characterized by a texture and by relative spatial and temporal arrangements of 
people, he lists them down as necessary conditions. 

Through the process of feedback, the system identifies the video clips relevant 
to the concept that the user is interested in. Now, since the system knows that texture 
and the spatial and temporal arrangements are necessary to the concept, it seeks to 
determine consistent patterns amongst the features deemed necessary, amongst the 
relevant videos. These patterns are then returned to the user, who is asked if they are 
consistent with the concept that he is searching for. If the user accepts these patterns 
as consistent with the concept, then they will be used to generate new query 
templates, as illustrated by Fig. 3. Including this new rule has two-fold impact on 
query template generation, namely it improves the speed of the search and increases 
the precision of the returned results. 

Generating Concept Covers 

The query defines a feature space where the search is executed. The feature 
space is defined by the attributes and relevancy weights of the visual template. In 
particular, the attributes define the axes of the feature space, and the relevancy 
weights stretch/compress the associated axes. Within this composite feature space, 
each video shot can be represented as a point in this space. The visual template 
covers a portion of this space. Since the visual template can differ in feature and in 
character (global against object level), the spaces that are defined by the templates 
differ and are non-overlapping. 

Selection of a few features may be insufficient to determine a concept, but it 
may be adequately represented by a suitable selection differing as to weight, for 
example. Thus, a concept can be mapped into a feature space. 

A concept is not limited to a single feature space nor to a single cluster. For 
example, with respect to the class of sunset video sequences, sunsets cannot be totally 
characterized by a single color or a single shape. Thus, it is important to determine 
not only the global static features and weights relating to a concept, but also those 
features and weights that can vary. 

The search for concepts starts by specifying certain global constants. Through 
a context questionnaire, the number of objects in the search is determined, and the 
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global features that are necessary to each object. These represent constraints in the 
search process that do not vary. 

A user gives an initial query specifying features and setting weights. A set 
intersection is taken with the set of necessary conditions defined by the user. The 
5 necessary conditions are left unchanged. Changes are made to the template based on 
changes to those features deemed sufficient. If the sets do not intersect, rules are 
derived that characterize the concept based on the necessary conditions and relevancy 
feedback. 

The relevancy weight of each feature indicates the tolerance that the user 
1 0 desires along each feature. This tolerance is then mapped to a distance threshold 
along each feature, e.g. d(o>) = l/(ao + c), defining a hyper-ellipsoid in the feature 
space searched in. The threshold determines the number of non-overlapping 
coverings possible. The number of coverings determines the size and number of 
jumps possible along that particular feature. The algorithm performs a breadth first 
1 5 search and is guided by three criteria: 

First, the greedy algorithm going in the direction of increasing recall: 
Compute all possible initial jumps. 
Convert each jump into the corresponding visual template. 
Execute the query and collate all the results. 
20 Show the results to the user for relevancy feedback and chose those 

results that maximize incremental recall as possible points of subsequent query. 

Second, a logarithmic search, as subsequent queries are searched by taking a 
smaller jump in a local region. The rationale is that the current query point produced 
good results and should be search carefully for further templates. Searching stops 
25 when enough visual templates have been generated to cover above 70 percent of the 
concept (i.e recall is above 70%). 

Third, as the breadth first search often yields too many possibilities to 
examine at once, feature level distributions are used to guide the search. The 
distribution along each feature has been pre-calculated. This information is used to 
30 select jumps to regions that have a high concentration of video shots and avoid sparse 
ones. 

Language Integration with SVT 

Known text based queries on images and videos rely on matching on 
keywords accompanying the image or the video. Keywords accompanying the data 
35 can either be generated manually or are obtained by association, i.e. keywords are 
extracted from the accompanying text (in the case of an image) or the captions that 
accompany videos. 

Such an approach precludes the possibility of any practical system containing 
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a very large database of videos or images for several reasons such as: 

It is not feasible to generate annotations manually for the existing database of 

videos. 

Most videos do not contain captions. 
5 There may not be an immediate correlation between the accompanying 

caption and the video. For example, during a baseball game the commentators may 
be talking about the exploits of Babe Ruth, who is not present in the game that is 
being played, and a text-based keyword on locating videos containing "Babe Ruth" 
will incorrectly display this video. 
1 0 Generating semantic content about a video by analysis of the video stream 

alone amounts to the computer vision problem which is known to be difficult. A 
more practicable approach is to simultaneously make use of the visual content such as 
the motion of objects, attributes like color and texture, with the descriptive power of 
natural languages. 

1 5 The user types in a string, which the system parses into a video model. 

VideoQ provides a "language" for inputting the query, in terms of a sketch. There is a 
simple correspondence between what exists in VideoQ and its natural language 
counterpart, as illustrated by Table 1. 

Attribute NLTvpe 

20 Motion Verb 

Color, Texture Adjective 

Shape Noun 

Spatial/temporal Preposition/Conjunction 

Table 1 

25 A constrained language set can be used, with a set of allowable words. A 

sentence is parsed into classes such as nouns, verbs, adjectives, and adverbs to 
generate a motion model of the video sequence. 

For example for the phrase "Bill walked slowly towards the sunset", the 
system can parse as shown in Table 2. 

30 Word NL-Tvpe 

Bill Noun 

Walked Verb 

Slowly Adverb 

Towards Preposition 

35 Sunset Noun 

Table 2 
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For verbs, adverbs, adjectives and prepositions, a small but fixed database can 
be used, as these are modifiers (or descriptors) on nouns (the objects). An noun (i.e. 
scenario/object) database may initially include a hundred scenes or so, and be 
extensible by user interaction. 
5 Each object may have a shape description that is modified by the various 

modifiers such as adjectives (color, texture), verbs (walked), adverbs (slowly). This 
can then be inserted into the VideoQ palette, where it may be subject to further 
refinement. 

When the parser encounters a word that is absent from its modifier 
1 0 database(i.e. the databases corresponding respectively to verbs, adverbs, prepositions, 
adjectives), it then looks up a thesaurus to determine if synonyms of that word are 
present in its database, and uses them instead. If that fails, it returns a message to 
indicate an invalid string. 

When the parser encounters a word that it cannot classify, the user must either 
1 5 modify the text or, if the word is a noun (like "Bill"), then he can indicate to the 

system the class (in this case a noun), and additionally indicate that the word refers to 
a human being. If the user indicates a noun that is absent from the system databases, 
then the user is prompted to draw that object in the sketch pad so that the system can 
learn about the object. In the database, attributes such as motion, color, texture and 
20 shapes can be generated at the object level, so that one level of matching can be at 
that level. 

As a further source of information, the audio stream can be used that 
accompanies a video, as illustrated by Fig. 4. Indeed, if the audio is closely correlated 
to the video, it may be the single most important source of the semantic content of the 
25 video. From the audio stream, a set of keywords can be generated, 10-20 per video 
sequence, for example. Then the search at the keyword level can be joined to the 
search that at the model level. Those videos can then be ranked the highest which 
match at the keyword (semantic) level as well as the motion-model level. 



EXAMPLES 

Semantic visual templates for retrieving video shots of slalom ski^rc 

1 . The system asks and the user answers questions regarding context. The 
semantic visual template is labeled "slalom". The query is specified as object-based, 
including two objects. 

2. The user sketches the initial query. The large blank background represents 
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the ski slope and the smaller foreground object the skier with its characteristic zigzag 
motion trail. 

3. Maximum relevancy weights are assigned to all the features associated 
with the background and the skier. The background feature is specified to remain 

5 static while those of the skier can vary during template generation. 

4. The system generates a set or test icons from which the user selects 
plausible feature variations in the skier's color and motion trajectory. 

5. The four selected colors and the three selected motion trails are joined to 
form 12 possible skiers. The list of skiers is joined with the single background, 

10 resulting in the 12 icons of Fig. 7 where groups of three adjacent icons are understood 
as having the same.color. 

6. The user chooses a candidate set to query the system. The system retrieves 
the 20 closest video shots. The user provides relevancy feedback to guide the system 
to a small set of exemplar for slalom skiers. 

15 Sunsets . A database was used which includes 72 sunsets in more than 1 952 

video shots. Using just an initial sketch, without semantic visual templates, a recall 
of 10 % and a precision of 35% were realized. Using semantic visual templates, 8 
icons were generated, with which 36 sunset were found, for a recall of 50% and a 
precision of 24%. 

20 High Jumpers . The database contains nine high jumpers in 2589 video shots. 

Without semantic visual templates, recall was 44% and precision 20%. With 
semantic visual templates recall was improved to 56% and precision to 25%. The 
system converged to a single icon different from the initial sketch provided by the 
user. 



WO 99/45483 



13 



PCT/US99/04776 



CLAIMS 

1 . A computerized method for generating a visual template for a concept, 
comprising the steps of: 

a. obtaining at least one initial query for the concept; 
5 b. generating at least one additional query related to the initial query; 

c. producing the additional query for inspection for appropriateness with 
respect to the concept; and 

d. in case of appropriateness, including the additional query in the visual 
template for the concept. 

10 2 * The method of claim 1 , wherein each query is represented by an 

icon/example image. 

3. The method of claim 1, wherein the initial query is obtained via a 
sketchpad. 

4. The method of claim 1, wherein generating the additional query comprises 
1 5 stepping a query feature with a step size which is inversely related to a weight 

associated with the query feature. 

5. The method of claim 1, wherein generating the additional icon comprises 
forming a join of plausible feature values. 

6. The method of claim 1, wherein appropriateness is ascertained by two-way 
20 user interaction. 

7. A computerized method for querying a video database for a concept using 
a subset of a natural language in conjunction with semantic visual templates, 
comprising the steps of: 

a. obtaining a textual query; 

25 b. parsing the query to generate visual attributes; 

c. using the visual attributes for forming a visual query; 

d. using the visual query to retrieve information; and 

e. displaying the information. 

8. The method of claim 7, wherein the textual query is obtained from a 
30 keyboard. 
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9. The method of claim 7, wherein the subset of the natural language 
comprises a small set of nouns, verbs, prepositions, adjectives and adverbs. 

10. The method of claim 7, further comprising a step of expanding the subset 
in an interactive fashion. 

1 1 . The method of claim 7, wherein parsing comprises the steps of: 

(i) establishing a correspondence between the query and the natural language 

subset; 

(ii) labeling different parts of the query as a noun, verb, adjective or 
preposition; and 

(iii) obtaining clarification if a word in the query is absent from the natural 
language subset and labeling the word accordingly. 

12. The method of claim 7, wherein forming the visual query comprises the 
step of establishing a correspondence between the natural language subset and a set of 
semantic visual templates generated by the method of claim 1, with the semantic 
visual template being a visual embodiment of a noun in the query, an adjective 
serving to modify the visual embodiment of the nouns, a verb serving to embody an 
action, and a preposition serving to establish a spatial and temporal order needed in 
forming a visual query. 

13. A computerized system for generating a visual template for a concept, 
comprising: 

a. means for obtaining at least one initial query for the concept; 

b. means for generating at least one additional query related to the initial 

query; 

c. means for producing the additional query for inspection for appropriateness 
with respect to the concept; and 

d. means in case of appropriateness for including the additional query in the 
visual template for the concept. 

14. The system of claim 1, wherein each query is represented by an 
icon/example image. 

15. The system of claim 1, wherein the means for obtaining the initial query 
comprises a sketchpad. 
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16. The system of claim 1 , wherein generating the additional query comprises 
stepping a query feature with a step size which is inversely related to a weight 
associated with the query feature. 

17. The system of claim 1, wherein generating the additional icon comprises 
forming a join of plausible feature values. 

18. The method of claim 1, wherein appropriateness is ascertained by two- 
way interaction between the system and a user. 

19. A computerized system for querying a video database for a concept using 
a subset of a natural language in conjunction with semantic visual templates, 
comprising: 

a. means for obtaining a textual query; 

b. means for parsing the query to generate visual attributes; 

c. means for using the visual attributes for forming a visual query; 

d. means for using the visual query to retrieve information; and 

e. means for displaying the information. 

20. The system of claim 19, wherein the means for obtaining the textual 
query comprises a keyboard. 

21. The system of claim 19, wherein the subset of the natural language 
comprises a small set of nouns, verbs, prepositions, adjectives and adverbs. 

22. The system of claim 19, further comprising means for expanding the 
subset in an interactive fashion. 

23. The method of claim 19, wherein the means for parsing comprises: 

(i) means for establishing a correspondence between the query and the natural 
language subset; 

(ii) mans for labeling different parts of the query as a noun, verb, adjective or 
preposition; and 

(iii) means for obtaining clarification if a word in the query is absent from the 
natural language subset and labeling the word accordingly. 



24. The system of claim 19, wherein the means for forming the visual 
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comprises means for establishing a correspondence between the natural language 
subset and a set of semantic visual templates generated by the system of claim 13, 
with the semantic visual template being a visual embodiment of a noun in the query, 
an adjective serving to modify the visual embodiment of the nouns, a verb serving to 
embody an action, and a preposition serving to establish a spatial and temporal order 
needed in forming a visual query. 



WO 99/45483 



PCT/US99/04776 



1/4 




QUERY 



RESULTS 



FEATURE JUMPS 




I 



FINAL 
ICON LIST 



FIG. 1 

SUBSTITUTE SHEET (RULE 26) 



WO 99/45483 



PCT/US99/04776 



2/4 




FIG. 2 



FEEDBACK 
QUERY ) m- ( RESULTS 




MODIFY 



COMPARE 



(^^LE ) -* 



GENERATE 




FIG. 3 



SUBSTITUTE SHEET (RULE 26) 



WO 99/45483 



PCT/US99/04776 



3/4 



LU 












"a! 


>- 


l- S 




2 uj 


< 


<»- 


q: 


S-f 


CD 


UJ < 


_l 


CO ZD 




CO 




> 
















UJ o 


EO 


o 

LU 


P o 


VID 


OBJI 


FEAT 
XTRA 






UJ 



o 
S< 

UJ 

o 



ZD 
CO 









o 


IDEO 


NTATI 


> 


LU 








o 




UJ 




CO 



_l 



UJ o 
< o 

E > 



SUBSTITUTE SHEET (RULE 26) 



WO 99/45483 



PCT/US99/04776 



4/4 



FIG. 5 







::V^Ir.;«;«iJV 




O 

















FIG. 6 



FIG. 7 



SUBSTITUTE SHEET (RULE 26) 



INTERNATIONAL SEARCH REPORT 



International application No. 
PCT/US99/04776 



A. CLASSIFICATION OF SUBJECT MATTER 

IPC(6) :G06F 17/30; 19/00 

US CL : 707/3,4,46,100,104 
According to International Patent Classification (IPC) or to both national classification and IPC 



B. FIELDS SEARCHED 
Minimum documentation searched (classification system followed by classification symbols) 
U.S. : 707/3,4,46,100,104 



Documentation searched other than minimum documentation to the extent that such documents 



are included in the fields searched 



Electronic data base exulted during the international search (name of data base and. where practicable, search terms used) 



APS. IS&R, EIC 

SEARCH TERMS: VISUAL, IMAGE, NATURAL LANGUAGE. RETRIEVE. SVT 



C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Category* 



Citation of document, with indication, where appropriate, of the relevant 



passages 



Relevant to claim No. 



US5.493.677 A (BALOGH et al.) 20 February 1996. col. 14. lines 



US 5,724,484 A (KAGAMI et al.) 03 March 1998, col. 8, lines 47- 
60. 

U S 5,708,805 A (OKAMOTO et al.) 13 January 1998. col. 1, lines 



1-24 



1-24 



1-24 



I Further documents are listed in the continuation of Box C. | | 



Special categories of cited documents: 



See patent family annex. 



document defining the general state of the art which ii not considered 
to be of particular relevance 

earlier document published on or after the international filing date " X * 
document which may throw doubti on priority claim(i) or which is 
cited to establish the publication date of another citation or other 
special reason (.as specified) m y 

document referring to an oral disclosure, use. exhibition or other 
means 

document published prior to the international filing date but later than . - 
the priority date claimed * 



Uter document published after the international filing date or priority 
date and not in conflict with the application but cited to understand 
the principle or theory underlying the invention 

document of panic alar relevance; the claimed invention cannot be 
considered novel or cannot be considered to involve an inventive step 
when the document is taken alone 

document of particular relevance; the claimed invention cannot be 
considered to involve an inventive step when the document is 
combined with one or more other such documents, such combination 
being obvious to a person skilled in the art 

document member of the same patent family 



Date of the actual completion of the international search 
29 APRIL 1999 



Date of mailing of the international search report 



1 * MAY 1999 



Name and madmg address of the ISA/ US 
Commissioner of Patents and Trademarks 
Box PCT 

Washington, D.C. 20231 
Facsimile No. (703) 308-5403 



Form PCT/ISA/2I0 (second sheet)(July 1992) , 



Authorized officer 

PAUL R. LINTZ 
Telephone No. (703) 305.4832 




BAKER BOTiS L.L.P. 



PATENT COOPERATION TREATY 



From the 

INTERNATIONAL PRELIMINARY EXAMINING AUTHORITY 



00 JUL -5 PM U- 57 

TO 



To: HENRY TANG 

BAKER & BOTTS, LLP 
30 ROCKEFELLER PLAZA 
NEW YORK NY 10112-0228 



PC^ 




NOTIFICATION OF TRANSMITTAL OF : 
INTERNATIONAL PRELIMINARY 
EXAMINATION REPORT 

(PCT Rule 71.1) 




Date of Mailing 

(day/ month /year) 



2 9 JUN 2000 



Applicant' s or agent' s file reference 
3 1596- PCT 


IMPORTANT NOTIFICATION 


International application No. 
PCT/US99/04776 


International filing date (day/ month /year) 
04 MARCH 1999 


Priority Date (day/ month /year) 
04 MARCH 1998 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK 



1 . The applicant is hereby notified that this International Preliminary Examining Authority transmits herewith the 
international preliminary examination report and its annexes, if any, established on the international application. 

2. A copy of the report and its annexes, if any, is being transmitted to the International Bureau for communication 
to all the elected Offices. 

3. Where required by any of the elected Offices, the International Bureau will prepare an English translation of 
the report (but not of any annexes) and will transmit such translation to those Offices. 

4 REMINDER 

The applicant must enter the national phase before each elected Office by performing certain acts (filing 
translations and paying national fees) within 30 months from the priority date (or later in some Offices) (Article 
39(l))(see also the reminder sent by the International Bureau with Form PCT/IB/301). 

Where a translation of the international application must be furnished to an elected Office, that translation must 
contain a translation of any annexes to the international preliminary examination report. It is the applicant's 
responsibility to prepare and furnish such translation directly to each elected Office concerned. 

For further details on the applicable time limits and requirements of the elected Offices, see Volume II of the 
PCT Applicant's Guide. 






Name and mailing address of the IPEA/US 

Commissioner of Patents and Trademarks 
Box PCT 

Washington, D.C. 2023 1 

Facsimile No. (703) 305-3230 


Authorized officer ^<^f7 • ^1 

KIM VU 

Telephone No. (703) 30S-6718 ^ 



Form PCT/IPEA/416 (July 1992)* 



PATENT COOPERATION TREATY 

PCT 

INTERNATIONAL PRELIMINARY EXAMINATION REPORT 
(PCT Article 36 and Rule 70) 



Applicant' s or agent* s file reference 
3 1596- PCT 


FOR FURTHER ACTION See Notification of Transmittal °* 1^™*°^ 

Preliminary Examination Report (Form PCT/IPEA/416) 


International application No. 
PCT/US99/04776 


International filing date (day 1 month/ year) Priority date (day/ month /year) 
04 MARCH 1999 04 MARCH 1998 


International Patent Classification (IPC) or national classification and IPC 
IPC(7): G06F 17/30; 19/00 and US CI.: 707/3,4,46,100.104 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK 



1. This international preliminary examination report has been prepared by this International Preliminary 
Examining Authority and is transmitted f> the applicant according to Article 36. 



2. This REPORT consists of a total of 

I I This report is also accompanied by 
' — ' been amended and are the basis for 
(see Rule 70.16 and Section 607 o, 

These annexes consist of a total of 




sheets. 



S, i.e., sheets of the description, claims and/or drawings which have 
report and/or sheets containing rectifications made before this Authority, 
te Adniinistrative Instructions under the PCT). 

sheets. 



3. This report contains indications relating to the following items: 
Basis of the report 



I 




II 


□ 


III 


□ 


IV 


□ 


V 




VI 


□ 


VII 


□ 


vm 


□ 



citations and explanations supporting such statement 
Certain documents cited 



Date of submission of the demand 
22 SEPTEMBER 1999 


Date of completion of this report 
25 MAY 2000 


Name and mailing address of the I PEA/ US 

Commissioner of Patents and Trademarks 
Box PCT 

Washington, DC 20231 

Facsimile No. (703) 305-3230 


Authorized officer^^""?} 

Telephone No. (703)^8-6718 r 



Form PCT/IPEA/409 (cover sheet) (July 1998) i 



INTERNATIONAL PRELIMINARY EXAMINATION REPORT 



International application No. 
PCT/US99/04776 



1. Basis of the report 



1. With regard to the elements of the international application: * 
| x | the international application as originally filed 
p^j the description: 



pages 
pages . 
pages . 



1-12 



NONE 



, as originally filed 

filed with the demand 



NONE 



, filed with the letter of 



| x| the claims: 

pages 

pages 

pages 

pages 



13-16 



NONE 



NONE 



, as originally filed 

as amended (together with any statement) under Article 19 
, filed with the demand 



NONE 



, filed with the letter of 



the drawings: 

pages 

pages 

pages 



1-3 



NONE 



, as originally filed 

, filed with the demand 



NONE 



| x| the sequence listing part of the description: 

pages NONE 

pages 



, filed with the letter of . 



NONE 



, as originally filed 

filed with the demand 



pages . 



NONE 



filed with the letter of 



2. With regard to the language, all the elements marked above were available or furnished to this Authority in the language in which 
the international application was filed, unless otherwise indicated under this item 

These elements were available or furnished to this Authority in the following language which is: 

| [ the language of a translation furnished for the purposes of international search (under Rule 23.1(b)). 
| [ the language of publication of the international application (tinder Rule 48.3(b)). 

| | the language of the translation furnished for the purposes of international preliminary examination (under Rules 55.2 and/ 
or 55.3). 

3 . With regard to any nucleotide and/or amino acid sequence disclosed in the international application, the international 
preliminary examination was carried out on the basis of the sequence listing: 

□ contained in the international application in printed form. 

| | filed together with the international application in computer readable form. 

| | furnished subsequently to this Authority in written form. 

[ [ furnished subsequently to this Authority in computer readable form. 

□ The statement that the subsequently furnished written sequence listing does not go beyond the disclosure in the 
international application as filed has been furnished. 

I I The statement that the information recorded in computer readable form is identical to the writen sequence listing has 
1 — 1 been furnished. 

4 | x| The amendments have resulted in the cancellation of: 



the description, pages_ 
the claims, Nos. 



NONE 



0 

Q 

I x| the drawings, sheets/fig NONE 



NONE 



5 • |x | This report has been drawn as if (some of) the amendments had not been made, since they have been considered to go 

beyond the disclosure as filed, as indicated in the Supplemental Box (RHe 70.2(c)).** 
* Replacement sheets which have been furnished to the receiving Office in response to an invitation under Atricle 14 are referred to 
in this report as "originally filed" and are not annexed to this report since they do not contain amendments (Rules 70.16 
and 70.17). 

**Any replacement sheet containing such amendments must be referred to under item 1 and annexed to this report. 

Form PCT/IPEA/409 (Box I) (July 1998)* 



INTERNATIONAL PRELIMINARY EXAMINATION REPORT 



International application No. 
PCT/US99/04776 



V. Reasoned statement under Article 35(2) with regard to novelty, inventive step or industrial applicability; 
citations and explanations supporting such statement 



1. statement 

Novelty (N) Claims 1-18 YES 

Claims 19-24 NO 

Inventive Step (IS) Claims NONE : YES 

Claims 1-24 NO 

Industrial Applicability (IA) Claims 1-24 YES 

Claims NONE NO 



2. citations and explanations (Rule 70.7) 

Claims 1-24 lack of inventive step under PCT Article 33(3) as being obvious over Balogh (US Pat. No. 5,493,677) in view 
of Okamoto (US Pat. No. 5.708,805). 

As per claims 1, 7, 13, and 19 Balogh discloses a method comprising of obtaining at least one initial query for the 
concept (see, col. 15, lines 12-14); 

generating at least one additional query related to the initial query (see, col. 3, lines 4-6); 

producing the additional query for inspection for appropriateness with respect to the concept (see, col. 10, lines 23-28); 

displaying the information (see, col. 17, lines 45-46). But, he does not specifically disclose a method of including the 
additional query in the visual template for the concept. However, Okamoto discloses a method of images contain more 
information and are more concrete or real than natural language, many images are employed in various documents (see, col. 
1, lines 11-13). 

It would have been obvious to a person of ordinary skill in the art at the time the Applicant's invention was made to 
modify the teachings of Balogh with including the additional query in the visual template for the concept as disclosed by 
Okamoto, because such modification would provide the system to the degree of similarity with the image retrieving information 
(see, col. 1, lines 54^56). 

As per claims 2, 5, 14, and 17 Balogh discloses a method wherein each query is represented by an icon/example image 
(see, figure 14, col. 17, lines 14-16). 

As per claims 3,8, 15, and 20 Balogh discloses a method wherein the textual query is obtained from a keyboard (see, 
cols. 13, and 17, lines 43-53, and 2-6). 

As per claims 4, 9, 16, and 21, 
the limitations of claims 4, 9, 16, and 21 have been mentioned in the analysis of claim 1 above. They are, therefore, lack of 
inventive step. 

(Continued on Supplemental Sheet.) 



Form PCT/IPEA/409 (Box V) (July 1998)* 



INTERNATIONAL PRELIMINARY EXAMINATION REPORT 


International application No. 
PCT/US99/04776 


Supplemental Box 

(To be used when the space in any of the preceding boxes is not sufficient) 


Continuation of: Boxes I - VIII 


Sheet 10 


I. BASIS OP REPORT: 




5. (Some) amendments are considered to go beyond the disclosure as filed: 
NONE 





V. 2. REASONED STATEMENTS - CITATIONS AND EXPLANATIONS (Continued): 



As per claims 6, 10, 18, and 22 Balogh discloses a method wherein appropriateness is ascertained by two-way 
interaction between the system and a user (see, abstract, lines 5-8). 

As per claims 11, and 23 Balogh discloses a method wherein establishing a correspondence between the query and 
the naturel language subset (see, col. 18, lines 38-45); 
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For database image/video retrieval, a semantic visual template (SVT) is a set of 
icons of example scenes/objects that characterize a concept, e.g. skiing, sunset and 
the like. SVT's provide for two-way interaction between a user and a system The 
user can provide the system with and initial sketch or example image, as a seed to 
the system to automatically generate other representations of the same concept The 
user then can pick those views for inclusion that are plausible for representating the 
concept. When an SVT has been established, the database can be searched with it 
for the user to provide relevancy feedback on the returned results. With established 
SVT's, the user can interact with the system at concept level. In forming new 
concepts, pre-existing SVT's can be used. A limited vocabulary can be parsed in 
conjunction with semantic visual templates for querying the system (figure 1) ' 
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